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Abstract 

This systematic review has analyzed 53 English language studies that assessed and compared 
peoples' self-reported and demonstrated information literacy (IL) skills. The objective was 
to collect empirical evidence on the existence of Dunning-Kruger Effect in the area of 
information literacy. The findings clearly show that this theory works in this area. It is 
concluded that there is no calibration in peoples' perceived and actual IL skills; in most cases 
low-performers overestimate their skills in self-assessments. The findings have theoretical 
and practical implications for librarians and IL educators. 

Keywords : information literacy; assessment; Dunning-Kruger Effect 


Mahmood, K. Do people overestimate their information literacy skills? A systematic review 
of empirical evidence on the Dunning-Kruger effect. Communications in Information 
Literacy, 10(2), 198-213. 

Copyright for articles published in Communications in Information Literacy is retained by the author(s). Author(s) also extend to Communications in 
Information Literacy the right to redistribute this article via other scholarly resources and bibliographic databases. This extension allows the authors' 
copyrighted content to be included in some databases that are distributed and maintained by for-profit companies. All other rights of redistribution 
are licensed by Communications in Information Literacy under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY- 
NC-SA4.0). 


[ Article ] 


Mahmood 

Do People Overestimate IL Skills 



200 


Communications in Information Literacy | Vol. 10, No. 2,2016 


Do People Overestimate their Information Literacy Skills? 
A Systematic Review of Empirical Evidence on the Dunning- 

Kruger Effect 


Introduction 

The assessment of self-perceptions has been one of the popular methods of evaluating 
information literacy skills of students and professionals. Self-efficacy, based on self¬ 
perceptions regarding particular behaviors, influences human functioning and is considered 
important for lifelong learning. Kurbanoglu, Akkoyunlu and Umay (2006) argued, "if 
individuals feel themselves competent and confident about the information literacy skills 
they will willingly undertake and easily solve information problems. Otherwise, it is more 
likely that they will avoid and hesitate to try solving information problems in their hands” 

(p. 731-732). From the point of view of educational psychology, self-assessment is a 
promising approach: "subjective ability is often considered a core belief that is the 
foundation of human motivation, performance accomplishments, and emotional well-being, 
and can therefore positively influence effort expenditure and task persistence, especially in 
the case of obstacles” (Rosman, Mayer & Krampen, 2015, p. 742). 

The literature on information literacy assessment repeatedly shows that self-reporting is not 
a substitute for the examination of people's actual information skills (Mahmood, 2013; 
Walsh, 2009). A major complaint against self-assessment is the lack of validity of this 
measure. The literature reported that people inflated their information skills: they were 
over confident in reporting their competencies compared to their actual performance. This 
behavior is referred to as Dunning-Kruger Effect. 

The Dunning-Kruger Effect was proposed by Justin Kruger and David Dunning of Cornell 
University. They conducted four studies in the areas of humor, logical reasoning, and 
English grammar, in which they asked participants to self-assess their ability and test 
performance. The studies found that the lowest performing participants overestimated their 
own abilities and believed that their abilities were above-average. On the other hand, the 
highest performing participants underestimated their abilities. It was also found that 
incompetent individuals were less likely to recognize competence or expertise in others 
(Folk, in press). Based on the findings, Kruger and Dunning (1999) argued that people who 
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were incompetent did not recognize their incompetence because they lacked metacognition, 
metamemory, metacomprehension, and self-monitoring skills. Dunning (2011) claimed that 
the poor performers face a double-curse: 

.. .because the skills or knowledge they need to produce a correct 
response are often the very same ones they need to judge the quality of 
that response. For example, the expertise needed to produce logically 
sound arguments is exactly the same knowledge needed to recognize 
whether a person has just made a logically sound argument. Generating 
a valid physics proof requires the same math skills needed to check the 
validity of the proof, (p. 152) 

Various studies substantiated the Dunning-Kruger Effect in many subject areas. For 
instance, the overestimating behavior was evident in university students' logical reasoning 
ability, specialist physicians' clinical practice, and salesmen's ability to sell (Hubka, 2015). 

The landmark study of Kruger and Dunning (1999) has been widely quoted by subsequent 
literature on the subject. 

In the area of information literacy, Gross and Latham (2007, 2009, 2012) replicated the 
research of Dunning and Kruger; in two out of three studies, they identified a disconnect 
between students' self-assessments of their information literacy skills and their actual skill 
level. Other studies also compared peoples' self-reported IL skills and their actual 
performance, but the existence of Dunning-Kruger Effect has been inconclusive. For 
example, Rosman, Mayer and Krampen (2015) were of the view that, "although a few 
studies have been conducted on the subject in information literacy research, no final 
conclusion on the validity of information literacy self-assessments can be drawn yet'' (p. 
743). 

The objective of the current study is to systematically collect and review the English 
language studies that provided empirical evidence for the existence or non-existence of 
Dunning-Kruger Effect in the assessment of peoples' information literacy skills. Specifically, 
the study addressed the following question: Does Dunning-Kruger Effect exist in studies 
that collected and compared peoples' perceived and actual information literacy skills? 
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Method 

This review was performed following the preferred reporting items for systematic reviews 
and meta-analyses (PRISMA) guidelines (Shamseer et al., 2015). 

Search strategy 

A systematic search of literature was carried out with the following search strategy: 

(“information skills” OR “information competencies” OR “information literacy” OR “library 
literacy” OR “library instruction” OR “bibliographic instruction”) 

AND 

(survey OR assessment OR evaluation OR test) 

A search was performed in two specialized databases—i.e., Library and Information Science 
Abstracts (LISA) and Library, Information Science & Technology Abstracts (LISTA)—and three 
general databases—i.e., Web of Science (WoS), Scopus, and Google Scholar— using the library 
portal at University of Dammam, Saudi Arabia in November 2015. The search was updated 
in March 2016. A manual search was also carried out based on review articles and a few key 
studies. It included a backward and forward citation search in Google Scholar. 

Inclusion and exclusion criteria 

The eligible studies for this review included those that assessed and compared peoples' self- 
reported information skills and their actual knowledge or skills. There was no limit applied 
for the year of publication, or for type of document. Therefore, journal articles, book 
chapters, conference papers, dissertations, reports, etc. were included. Studies reporting 
assessment of all types of participants (students, professionals, etc.) were included. However, 
only English language studies were selected. 
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Figure 1 - Four-phase flow diagram of the selection procedure of studies 



Study selection and data extraction 

The flow chart in Figure 1 shows the process of screening and selection of eligible studies 
and reasons for exclusion. The two stages of screening—i.e., title/abstract and full text- 
resulted in the selection of 53 studies for inclusion in this review. A data extraction form 
was completed for each eligible study to collect information on the name of first author, 
publication year, characteristics of sample, method of assessing self-reported skills, method 
of assessing actual skills, and findings regarding Dunning-Kruger Effect. 
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Results 

Overview of studies 

A search in five databases and a manual search yielded a total of 2,245 documents, of which 
the author collected and scanned 209 full text documents after an initial scanning of titles 
and abstracts. Fifty-three studies met the inclusion criteria with results of comparison 
between participants’ self-reported information skills and their actual knowledge or skills. 
Table 1 ( provided as a supplement to this article ) shows a summary of the data extracted 
from the selected studies. The year of publications ranged between 1986 and 2015. Most of 
the researches were published in library and information science journals, but some were 
published in the literature of other fields. Thirty-three studies were conducted in USA. 
Research conducted in other countries included UK (7 studies), Australia (5 studies), Canada 
(2 studies), and one study each in New Zealand, Belgium, Malaysia, Norway, Germany and 
the Netherlands. The participants in 50 studies were current students. Participants of the 
remaining three studies were medical practitioners; volunteers from financial, retail, 
consulting, and distribution sectors; and past students. In most cases, the participants were 
undergraduates, and in many cases they were newly admitted undergraduate students. 
Participants belonged to various subject areas of pure, applied, and social sciences; 
represented fields included medicine, nursing, biology, psychology, engineering, business, 
law, education, and management information systems. The sample sizes ranged between 15 
and 2,114 participants. 

The studies under review used various methods to assess self-reported information skills of 
the participants. The most popular was a questionnaire survey. In most cases, the 
researchers used locally designed questionnaires. In some studies, well-known instruments 
were used: e.g., the Research Readiness Self-Assessment (RRSA) tool, the Information 
Literacy Self-Efficacy scale, etc. A few studies used interviews to collect such data. 
Interestingly, there is more variety in the methods of assessing actual knowledge or skills of 
the participants. These methods include open-ended paper-and-pencil and online tests, 
multiple choice question (MCQ)-based tests/questionnaires, observations, field notes, 
participant reflexive journals, expert grading of searching tasks, analysis of students’ theses, 
quizzes, and information related assignments. 

Evidence on Dunning-Kruger Effect 

In 34 studies (64%) the evidence clearly showed that the participants overestimated their 
self-reported IL skills compared to their actual skills. In seven other studies (13%) a partial 
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evidence of overconfidence was available. On the other hand, two studies (4 %) fully and 
three studies (6 %) partially provided evidence that the participants underestimated their 
skills. Only four studies (8%) showed a positive correlation between perceived and actual 
skills, while another group of four studies (8%) showed this correlation partially. This means 
there was a match between two types of assessment. Five studies (9%) found weak 
(statistically insignificant) or no correlation between self-reported and actual skills. The 
direction of discrepancy (over or under) was not evident in these cases. An overall evidence 
of inconsistency (overestimated, underestimated, weak correlation, no correlation) was 
found in 49 (full or partial) out of 53 studies (92%). 

Medicine (an applied science) and business (a social science) were the largest subject groups 
in this review. A subject analysis of the evidence of Dunning-Kruger Effect shows that this 
behavior was found in six of nine medicine-related studies while it was found in seven of 
eight business-related studies. Based on this finding, we may infer that the overestimation 
behavior was greater in social science participants. Overestimation behavior was found in 
31 of 37 studies on undergraduates; conversely, only one study on graduate students fully 
and another partially reported overestimation of IL skills. From this, we may infer that the 
higher academic level positively influences the accuracy of students' assessment of their own 
IL skills. 

Discussion 

The current study is the first of its kind in the area of information literacy. The results based 
on 53 studies clearly reveal that the Dunning-Kruger Effect exists as it relates to peoples' 
information literacy skills. The analyzed studies came from a variety of geographical and 
subject areas and were conducted on different age groups. The findings substantiate the 
notion that people generally inflate their perceived level of skills in a particular domain. 
There is no match between self-efficacy and actual performance. Ninety-two percent of the 
cases presented should be considered as ample evidence to conclude the debate on Dunning- 
Kruger Effect in the area of information skills. 

Collection and analysis of a large amount of evidence in this study may have implications for 
both theory and practice. The first implication is that these findings pose an important 
question on the validity of self-assessment of IL skills. Latham and Gross (2008) have 
already suggested that "in terms of information literacy, confidence is not a reliable 
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predictor of competence” (p. 11). Information literacy practitioners may include subjective 
approaches to their test batteries, but they cannot rely solely on these measures. 

Another implication of Dunning-Kruger Effect is that the “individuals with below- 


proficient skills are unlikely to seek remediation for skills they believe they have, and that 


because they have a high level of confidence in their ability, they are unlikely to seek help, 
even when their attempts at finding information fail or result in low-quality information or 
partial answers” (Gross & Latham, 2012, p. 574). They will not have any motivation to join 
IL training, or they may be disengaged from classes. 

There is a general perception in society that the “Google generation,” with their computing 
and Internet skills, are also information literate. The research on this subject has found that 
“this assertion was a 'dangerous myth/ Digital familiarity and information literacy could not 
be equated, and there was no apparent evidence of an improvement (or particularly of a 
deterioration) of young people's information skills” (Bates, 2013, p. 176). They must be 
taught such skills. 

Findings of the current study warrant further research in the area of IL assessment. 
Specifically, preliminary indications of differences in the Dunning-Kruger Effect between 
applied sciences and social sciences, and between undergraduates and graduates, lead to 
formulate hypotheses to be tested. The author is suggesting here a few questions to be 
answered in future studies: 

1. Which IL assessment techniques are being used? In which frequency? With how 
much accuracy of measuring actual IL skills? 

2. What is the difference in the magnitude of Dunning-Kruger Effect in different 
cohorts based on age, gender, subject area, academic level, profession, cultural 
background, and frequency of already attended IL training programs? 

3. Is the level of Dunning-Kruger Effect different in various components of 
information literacy? 

4. Are librarians and IL decision-makers aware of the problem of Dunning-Kruger 
Effect? To what extent they consider this phenomenon in planning IL programs? 

5. What special measures IL program planners take to motivate and involve low- 
achievers' of IL skills in training programs? 
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Limitations 

Although this systematic review was rigorous, the search was limited to the English 
language studies. There may be informative studies published in other languages that were 
missed in this review. Two major abstracting databases from the library and information 
science field and three general indexing and citation databases were used, and a large 
number of documents were scanned to maximize the search; however, some relevant 
studies may have been overlooked. Furthermore, the nature of systematic reviews like this 
reflects the shortcomings of the studies reviewed; included studies were heterogeneous in 
terms of the methods of data collection and analysis. Various definitions of information 
literacy skills were used: i.e., ACRL or other standards, IT skills, ICT skills, basic computing 
skills, assignment writing skills, research skills, etcetera. Terms of “knowledge” and “skill” 
were sometimes used differently and sometimes interchangeably. Similarly, overestimating 
of self-assessment was reported differently in studies: i.e., overconfidence, high level of 
comfort, over-assumption of performance in IL test, overrating the quality of searching 
techniques/sources found, et cetera. 

Conclusion 

This systematic review analyzed studies that assessed peoples' self-reported and actual 
information literacy skills. The objective was to collect empirical evidence on the existence 
of Dunning-Kruger Effect. The findings clearly show that this theory can be applied in the 
area of information literacy. Based on the results, we can conclude with more confidence 
that there is little calibration in peoples' perceived and actual IL skills. In most of the cases, 
low-performers overestimate their skills in self-assessments. These findings have 
implications for theory and practice. As self-efficacy scales are not reliable instruments to 
assess IL skills, knowledge and skill tests and practical assignments may be used for the 
important task of IL assessment. Librarians and IL educators should also design training 
programs to look after the needs of low-achievers. Further studies are also needed to 
explore the existence and impact of Dunning-Kruger Effect in different cohorts. 
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